1. Field of the Invention
The present invention relates to direct data placement protocol.
2. Description of Related Art
In transmission control protocol/Internet protocol (TCP/IP), data travels between nodes on a computer network in pieces called segments or packets. The sending node divides the data into segments, sends the packets across the network to the receiving node, and the receiving node reassembles the received packets and stores the data in memory (or a memory buffer). The data carried by TCP segments (the payload) is divided into units called protocol data units (PDU's), or framed protocol data units (FPDU's). Each FPDU includes a header. The header contains control and placement fields that define the final placement location for the TCP payload carried in a TCP segment.
Typical communications over TCP/IP require copy operations, which add latency and consume significant CPU and memory resources. Direct data placement protocol (DDP) enables an upper layer protocol (ULP) to send data to a data sink without requiring the data sink to place the data in an intermediate buffer. When the data arrives at the data sink, the network interface places the data directly into the ULP's buffer. This procedure enables the data sink to consume substantially less memory bandwidth than a buffered model, because the data sink is not required to move the data from the intermediate buffer to the final destination. Additionally, this procedure enables the network protocol to consume substantially fewer CPU cycles compared to using the CPU to move the data. This procedure also removes any bandwidth limitations caused by slow data copying speeds of the CPU. The DDP specification (“Direct Data Placement over Reliable Transports,” available from The Internet Engineering Task Force (IETF) at http://tools.ietf.org/html/rfc5041) provides further information on DDP. The DDP specification is incorporated herein by reference.
DDP supports two basic data transfer models—a tagged buffer data transfer model and an untagged buffer data transfer model. The tagged buffer data transfer model requires the data sink to send the data source an identifier for the ULP buffer, referred to as a steering tag (STag). A ULP-defined method transfers the Slag to the data source. Once the data source ULP has an STag for a destination ULP buffer, it can request that DDP send the ULP data to the destination ULP buffer by specifying the STag to DDP. FIG. 1 illustrates a tagged buffer DDP header 20, including an STag 22.
The untagged buffer data transfer model enables data transfer without requiring the data sink to advertise a ULP buffer to the data source. The data sink can queue up a series of receive ULP buffers. An untagged DDP message from the data source consumes an untagged buffer at the data sink. FIG. 2 illustrates an untagged buffer DDP header 24.
At the data source, the DDP layer segments the data contained in a ULP message into a series of DDP segments/TCP segments (used interchangeably herein), where each DDP segment contains a DDP header and ULP payload. DDP message segmentation at the data source is accomplished by identifying a DDP message (which corresponds one-to-one with a ULP message) uniquely and then, for each associated DDP segment of a DDP message, by specifying an octet offset for the portion of the ULP message contained in the DDP segment.
The remote direct memory access protocol (RDMAP) is layered on top of DDP and uses the two buffer models available from DDP. FIG. 3 depicts the relationship between upper layer protocols (ULPs) 26, RDMAP 28, DDP protocol 30, the framing layer 32, and the transport 34. If RDMAP is layered over DDP/MPA/TCP, then the respective headers 36 and ULP Payload 38 are arranged as shown in FIG. 4. The RDMAP specification (“A Remote Direct Memory Access Protocol Specification,” available from IETF at http://tools.ietf.org/html/rfc5040) provides further information on RDMAP. The RDMAP specification is incorporated herein by reference.
Many current protocols used in Internet applications and elsewhere assume that data is delivered and placed in order. RDMAP, by contrast, does not provide ordering among messages on different RDMAP streams. TCP segments may thus arrive out of order at the data sink. Further. FPDU's do not always align with the payloads of their respective TCP segments, meaning that the first word of the payload may not be the FPDU header. In these situations, the receiving node must locate the FPDU header within the payload in order to properly place the data in memory.
Marker PDU aligned framing (MPA) is one method for locating FPDU headers in segments delivered out of order. MPA works as an adaptation layer between TCP and DDP. MPA markers identify the start of FPDU's when packets are received out of order. A marker is a back pointer to the previous FPDU header. The markers are located at fixed intervals in the data stream so that the receiving node can easily find them. The MPA specification (“Marker PDU Aligned Framing for TCP Specification,” available from IETF at http://tools.ietforg/html/rfc5044) provides further information on MPA. The MPA specification is incorporated herein by reference.
Unfortunately, the process of inserting markers into the data stream prior to data transmission is cumbersome. Further, the process of extracting the markers from the data stream prior to data storage is also difficult. Using markers thus increases hardware complexity, since extra computing power is needed to deal with the cumbersome processes of adding and extracting markers.